Papers with DROP dataset
Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models (2023.starsem-1)
Copied to clipboard
| Challenge: | Recent work suggests that pretrained language models perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. |
| Approach: | They propose an extended pretraining approach that addresses both in one extended step . they propose a novel extended pre training objective called Inferable Number Prediction Task to improve numeracy. |
| Outcome: | The proposed approach improves reading comprehension and inference-on-tables tasks without architectural changes or pretraining from scratch. |
NumNet: Machine Reading Comprehension with Numerical Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing numerical MRC models are weak in numerical reasoning, such as addition, subtraction, sorting and counting. |
| Approach: | They propose a numerical MRC model that integrates numerical reasoning into existing MRC models and achieves an EM-score of 64.56% on the DROP dataset. |
| Outcome: | The proposed model outperforms all existing machine reading comprehension models by considering the numerical relations among numbers on the DROP dataset. |
Paired Examples as Indirect Supervision in Latent Decision Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new method to learn compositional structured models is needed . end-task supervision provides only a weak indirect signal on values the latent decisions should take. |
| Approach: | They propose a way to leverage paired examples that provide stronger cues for learning latent decisions . they use a DROP dataset to acquire paired questions that provide strong cue signals . |
| Outcome: | The proposed approach improves compositional question answering on a DROP dataset. |
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks. |
| Approach: | They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks. |
| Outcome: | The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy. |